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1  Introduction 


Many  user  interface  management  systems  (UIMSs)  have  a  library  of  interface  building 
blocks  that  can  be  used  to  assemble  the  interface  for  a  program  [Tanner83].  An 
intelligent  UlMS  has  a  reasoning  component  that  determines  how  to  use  this  library 
based  on  a  model  of  the  program,  the  user  and  interface  design.  For  instance,  the 
Integrated  Interfaces  system  [Arens8S]  uses  a  model  of  the  objects  and  operations  in 
the  program,  and  a  model  of  interface  building  blocks  such  as  icons,  menus,  and  maps. 
The  reasoning  component  uses  a  set  of  rules  to  choose  what  interface  building  blocks 
to  use,  and  then  connects  them  with  the  objects  and  operations  of  the  application. 

Connecting  the  interface  building  blocks  with  different  application  requires  that 
the  building  blocks  use  a  standard  language  to  communicate  with  the  application 
objects  and  operations.  Otherwise,  it  is  necessary  to  write  a  program  to  translate 
between  the  languages  used  by  the  building  blocks  and  the  application,  making  it 
impossible  to  automatically  connect  building  blocks  and  applications. 

Suppose,  for  instance,  that  applications  are  programmed  using  an  object-oriented 
programming  language.  The  application  objects  would  be  defined  as  classes,  and 
the  application  operations  would  be  defined  as  methods  for  those  classes.  In  such 
an  environment,  the  standard  language  for  user  interface/application  communication 
would  consist  of  a  list  of  methods  that  each  application  object  must  provide.  If 
each  application  object  provides  these  methods,  then  the  interface  building  blocks 
could  communicate  with  any  application  object  since  the  building  blocks  would  use 
only  these  methods  to  communicate  with  the  application.  Such  a  standard  language 
would  make  the  building  blocks  plug-compatible  with  a  large  number  of  application 
programs. 

Defining  such  a  standard  language  is  difficult  because  the  language  must  support 
a  wide  variety  of  interface  styles  and  a  wide  variety  of  applications  programs.  This 
generality  requirement  generates  two  conflicting  goals: 

•  The  language  must  define  the  communication  with  the  application  in  abstract 
terms,  without  referring  to  any  particular  interface  style.  Otherwise,  the  appli¬ 
cation  will  not  be  independent  of  interface  style,  and  hence  cannot  support  a 
variety  of  interfaces. 


•  The  language  must  support  the  communication  of  the  information  needed  for 
each  interface  style,  including  the  information  needed  to  support  application- 
dependent  feedback  (semantic  feedback).  Since  different  interaction  styles  pro¬ 
vide  different  kinds  of  feedback  ( e.g .  rubberbanding,  gravity,  highlighting),  it 
appears  that  the  language  must  be  tuned  to  particular  interface  styles. 

This  paper  describes  a  language  that  achieves  a  good  compromise  between  these  two 
conflicting  goals.  To  achieve  the  generality  goal,  the  language  is  based  on  an  ex¬ 
tension  of  the  language  new  of  programs  [Foley82,  MoranSl,  Newman79],  a  general 
model  of  interactive  programs.  The  extensions  involve  mainly  enriching  the  descrip¬ 
tion  of  the  semantics  of  a  program.  To  ensure  that  the  language  supports  a  wride 
variety  of  interface  styles,  the  language  primitives  were  tested  with  a  large  number 
of  interaction  styles,  and  refined  to  make  sure  that  the  appropriate  information  can 
be  communicated.  The  resulting  language  primitives  allow  the  communication  of  the 
application-specific  information  needed  in  many  interface  styles,  but  the  information 
is  described  at  a  high  enough  level  of  abstraction  so  that  the  details  of  the  interface 
styles  remain  hidden. 

The  paper  is  organized  as  follows.  Section  2  presents  the  language  view  of  pro¬ 
grams.  Section  3  introduces  the  primitives  of  the  proposed  language,  and  section  4 
shows  how  these  primitives  can  explain  the  behavior  of  the  interface  to  a  chess  pro¬ 
gram.  Section  5  describes  Nephew,  a  UIMS  based  on  the  proposed  communication 
language,  and  shows  how  the  interface  to  the  chess  program  is  implemented  with 
Nephew.  Finally,  section  C  describes  how  the  language  can  be  used  in  the  reasoning 
component  of  intelligent  interfaces. 

2  The  Language  View  of  Programs 

Communication  is  "the  exchange  of  meanings  between  individuals  through  a  common 
system  of  symbols"  [EncyclopaeT  I].  To  communicate  a  meaning,  called  a  concept  in 
this  paper,  the  sender  must  encode  the  concept  into  a  sequence  of  symbols  with  a 
concrete  physical  representation  that  r,o>  hr  prrrrivr  t  |v  ; l,r  r(<(river.  Th.  n-ceivc 
must  decode  the  symbols  it  perceives,  and  extract  the  corresponding  concept. 

A  human  and  a  program  communicate  concepts  by  encoding  them  as  changes  in 
the  state  of  input  and  output  devices  attached  to  the  computer.  1'sers  communicate 
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concepts  to  programs  by  manipulating  the  input  devices.  Programs  monitor  the  state 
of  the  devices  to  recover  the  concepts,  process  them,  and  produce  new  concepts,  which 
are  transmitted  to  the  user  by  encoding  them  in  the  state  of  the  output  devices. 

The  primitives  of  the  proposed  language  were  identified  by  classifying  the  concepts 
that  users  and  programs  communicate  via  graphical  user  interfaces.  Each  concept  cat¬ 
egory  corresponds  to  a  primitive  of  the  language.  Put  in  object-oriented  terms,  each 
concept  category  corresponds  to  a  method  that  computes  the  information  embodied 
in  the  concept. 

The  classification  of  the  concepts  that  users  and  programs  communicate,  called 
communication  concepts ,  is  based  on  the  language  view  of  programs  [Foley82,  MoranSl, 
Newman79].  According  to  the  language  view,  the  language  to  communicate  with  a 
program  has  four  levels  called  the  conceptual,  semantic,  syntactic  and  lexical  levels. 
The  conceptual  level  describes  the  tasks  the  user  is  able  to  accomplish  using  the 
program.  Tasks  are  specified  by  decomposing  them  into  subtasks,  and  subtasks  are 
decomposed  further  until  they  can  be  accomplished  using  a  single  operation  provided 
by  the  program.  The  semantic  level  describes  these  operations  and  the  objects  that 
they  operate  on.  The  syntactic  and  lexical  levels  describe  how  the  user  accesses  the 
objects  and  operations  using  the  input  and  output  devices. 

These  levels  are  consistent  with  the  definition  of  communication  given  at  the 
beginning  of  this  section.  The  communication  concepts  are  the  meanings  encoded 
in  the  lexical  and  syntactical  levels.  The  user  transmits  them  in  order  to  use  the 
facilities  provided  by  the  program,  i.e.  the  objects  and  operations. 

Even  though  the  division  into  levels  appears  intuitive,  no  precise  notation  for 
describing  the  levels  of  an  application  program  exists.  Moran’s  CLG  [MoranSl]  is 
perhaps  the  most  complete  notation,  but  it  is  not  precise  enough  to  be  executable, 
and  many  aspects  of  graphics-based  interfaces  cannot  be  expressed  in  it.  Most  of 
the  work  on  notations  for  describing  programs  according  to  the  language  model  has 
concentrated  on  the  syntactic  and  lexical  levels.  BNF,  transition  networks,  and  event 
based  systems  are  the  most  popular  methods  [Green86]. 

The  specification  of  the  semantic  level  usually  consists  of  a  list  of  type  and  proce¬ 
dure  declarations  like  those  used  in  MIKE  [01sen86]  and  the  ones  used  in  Foley’s  ct. 
al.  user  interface  design  environment  [Foley88].  The  idea  behind  such  specifications 
is  that  the  types  specify  the  objects  that  the  application  provides,  and  the  procedures 
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specify  actions  that  can  be  performed  on  those  objects. 

This  kind  of  specification  does  not  capture  many  attributes  of  a  program  that  are 
conveyed  to  the  user  via  graphical  user  interfaces.  For  instance,  the  user  interface  for 
a  chess  program  could  highlight  the  pieces  that  can  be  legally  moved  at  any  given 
moment.  To  highlight  the  pieces  the  user  interface  has  to  somehow  find  out  which 
pieces  can  be  moved.  Some  supporters  of  the  type/procedure  method  of  specifica¬ 
tion  would  say  that  one  could  simply  include  in  the  specification  a  procedure  that 
computes  the  relevant  information.  This  is  a  poor  solution  because  such  a  specifica¬ 
tion  does  not  record  the  purpose  of  the  procedure,  which  is  to  test  whether  a  given 
piece  is  a  legal  parameter  for  the  move  operation.  This  specification  would  make  it 
impossible  to  construct  a  generic  user  interface  building  block  that  highlights  objects 
that  are  correct  parameters  to  an  operation.  The  building  block  would  not  know 
which  procedure  to  call,  and  even  if  it  knew,  it  would  not  know  what  parameters  to 
provide  and  in  what  order.  The  specification  language  should  capture  more  about 
the  meaning  of  the  types  and  procedures.  The  language  should  allow  one  to  express 
that  the  “test -chess- piece”  procedure  tests  whether  an  object  is  a  legal  candidate  to 
become  an  operation  input.  Any  interface  building  block  that  needs  this  information 
can  get  it  by  using  the  appropriate  language  construct. 

Some  UIMSs  [Hayes85,  SmithS4]  specify  the  semantics  of  programs  using  for¬ 
malisms  that  specify  more  than  types  and  procedures,  and  capture  the  information 
needed  to  support  the  generation  of  menu-based  interfaces.  Unfortunately,  they  do 
not  capture  some  of  the  aspects  of  the  semantics  of  a  program  needed  to  support 
highly  interactive  graphical  user  interfaces. 

The  difficulties  with  the  specification  of  the  levels  of  the  language  model  might 
suggest  that  the  model  is  inappropriate  [I\amran83].  This  paper  shows  that  many 
of  the  difficulties  of  the  language  model  can  be  overcome  by  enriching  the  semantic 
level  description. 

Finally,  the  classification  of  the  concepts  that  users  and  programs  communicate 
is  also  based  on  an  analysis  of  the  user  interface  of  many  Macintosh  programs  {e.g. 
M  acDraw.  Excel  and  Finder).  The  communication  concepts  can  explain  the  behavior 
of  the  interface  features  of  these  programs. 

The  communication  concepts  introduced  in  the  next  section  capture  the  distinc¬ 
tions  in  semantics  that  are  relevant  to  the  construction  of  graphical  user  interfaces. 
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Each  class  of  communication  concept  captures  a  different  semantic  distinction. 

3  Communication  Concepts 

The  communication  concepts  are  divided  into  input  and  output  communication  con¬ 
cepts.  The  input  communication  concepts  are  the  meanings  encoded  in  the  gestures 
produced  with  the  input  devices  (syntactic  and  lexical  levels).  These  concepts  express 
what  the  user  wants  to  do  with  the  objects  and  operations  (semantic  level). 

The  set  of  concepts  that  a  user  might  want  to  express  about  the  objects  and 
operations  of  a  program  are  open  ended.  The  power  of  the  set  of  communication 
concepts  presented  below  is  that  the  set  is  small,  yet  it  is  rich  enough  to  explain  the 
behavior  of  a  large  variety  of  graphical  user  interfaces.  The  set  of  interfaces  includes 
mouse  based  systems  such  as  the  Macintosh  interface.  The  communication  concepts 
can  explain  the  effect  of  every  input  event,  every  mouse  movement,  for  a  large  variety 
of  interfaces.  Section  4  illustrates  this  using  a  chess  program. 

The  input  communication  concepts  are  used  by  the  user  to  communicate  to  a 
program: 

Activate  operation :  expresses  the  user’s  intention  to  invoke  an  operation.  After  acti¬ 
vation  the  user  performs  other  activities,  which  are  described  below,  and  then 
executes  the  operation. 

Execute  operation :  requests  that  an  operation  be  executed.  If  the  operation  is  ready 
to  be  executed  ( e.g .  all  required  inputs  have  been  specified),  the  appropriate 
procedure  is  called.  Otherwise,  a  communication  concept  is  transmitted  to  tell 
the  user  about  the  error. 

Bind  input  of  operation  to  value:  specifies  a  value  for  an  input. 

Preview  operation:  computes  an  approximation  of  the  effects  that  executing  the  op¬ 
eration  would  have,  given  the  current  setting  of  the  input  values.  The  rubber¬ 
banding  effect  in  many  graphical  operations  is  an  instance  of  the  use  of  the 
preview  concept. 

Abort  operation:  requests  that  the  execution  or  activation  of  an  operation  be  termi¬ 
nated. 
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These  communication  concepts  can  be  qualified  with  the  qualifiers  plan  and  not-plan. 
Where  as  the  bind  input  to  value  concept  specifies  the  value  of  an  input  to  an  operation, 
the  plan  bind  input  to  value  concept  tells  the  program  that  the  user  has  entered  a  state 
in  which  he  or  she  can  specify  the  given  input.  For  instance,  moving  the  mouse  over 
a  check  box  icon  can  be  interpreted  as  a  plan  to  set  a  value.  Clicking  the  mouse  will 
set  the  value,  but  moving  the  mouse  away  from  the  check  box  without  clicking  will 
not  set  the  value,  and  can  be  interpreted  as  a  not-plan  bind  input  concept.  Section  4 
illustrates  how  these  qualifiers  are  used  in  the  interface  for  a  chess  program. 

The  output  communication  concepts  are  used  by  the  program  to  communicate  to 
the  user: 

Contents,  the  state  of  an  object. 

Changes:  a  change  in  the  contents  of  an  object. 

The  following  set  of  output  communication  concepts  refers  only  to  operations: 

Alternatives:  the  set  of  values  from  which  an  input  for  an  operation  must  be  chosen. 

Correct  input  of  operation:  the  value  true  if  the  input  of  an  operation  is  bound  to  a 
valid  value:  otherwise,  a  description  of  why  the  binding  is  incorrect. 

Using  object  as  input  of  operation:  a  concept  that  specifies  that  a  given  object  is  be¬ 
ing  used  as  an  input  value  for  an  operation. 

More  details  about  the  meaning  of  the  communication  concepts,  and  the  role  they 
play  in  the  communication  of  a  user  with  a  program  can  be  found  in  the  author’s 
thesis  [SzekelySS]. 

These  communication  concepts  are  the  categories  in  a  classification.  To  precisely 
describe  a  program  it  is  often  necessary  to  define  specializations  of  some  of  these 
concepts.  For  instance,  the  changes  communication  concept  reports  that  an  object 
has  changed,  but  does  not  specify  how.  A  specialization  of  the  changes  concept  should 
be  used  when  it  is  desired  to  specify  how  an  object  changes. 

For  example,  a  chess  program  could  transmit  the  PIECE  clump**  communication 
concept  when  PIECE  is  moved  from  one  location  to  another,  or  when  PIECE  is  taken. 
The  changes  communication  concept  could  be  specialized  for  the  chess  program  into, 
say,  changes-moved  and  changes-taken  to  distinguish  between  the  two  kinds  of  changes. 
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These  two  concepts  would  allow  the  chess  program  to  transmit  more  accurate  infor¬ 
mation  about  changes. 

The  most  important  point  about  the  communication  concepts  is  that  they  can 
express  the  communication  between  a  user  and  a  program  in  abstract  terms,  without 
reference  to  part  icular  interaction  techniques.  The  interpret  at  ion  of  every  input  event . 
and  all  display  updates  can  lie  viewed  as  the  encoding  of  one  of  the  communication 
concepts. 

4  An  Example 

This  section  illustrates  how  t  he  communicat  ion  concepts  can  explain  the  input  /output 
behavior  of  a  program.  The  figures  that  follow  show  snapshots  of  a  chess  program 
display  while  the  user  drags  a  piece  using  the  mouse.  The  interface  behaves  as  follows: 
when  t he  user  presses  the  mouse  button  over  a  piece  that  can  be  moved,  the  piece  is 
highlighted,  then  an  outline  of  the  piece  follows  the  rr.^^ae.  The  user  can  then  drag  the 
piece  to  its  destination  and  release  the  mouse  button  to  drop  the  piece  there.  While 
dragging,  the  location  under  the  mouse  highlights  when  it  is  a  legal  destination  for 
the  piece. 

The  following  screen  snapshot  shows  the  chess  board  after  the  user  moves  the 
mouse  over  the  knight,  but  has  not  pressed  the  mouse  button. 


The  following  snapshots  consist  of  two  parts.  The  first  part  describes  an  input 
event  and  its  related  communication  concepts,  and  the  second  one  shows  the  new 
screen  state  after  the  communication  concepts  arc  processed.  In  what  follows  the 
erent  describes  the  input  event  received  by  the  program,  the  input  CC  is  the  input 
communication  concept  encoded  in  the  event,  the  output  CC  is  the  output  communi¬ 
cation  concept  produced  as  a  result  of  processing  the  input  CC,  and  the  display  is  a 
description  of  how  the  program  displays  the  output  CC. 


Event:  the  user  presses  the  mouse  button  over  the  knight. 

Input  CC:  bind  the  piece  input  of  the  move  operation  to  piece-under-moese. 
Output  CC:  the  PIECE  input  of  the  MOVE  operation  is  correct. 

Display:  highlight  the  piece. 


Event:  user  moves  the  mouse  to  adjacent  square. 

Input  CC:  plan  to  bind  the  LOCATION  input  of  the  MOVE  operation 

to  LOCATION  -  U  N  DER-MOt'SE. 

Output  CC:  the  LOCATION  input  of  the  MOVE  operation  is  incorrect. 
Display:  nil,  only  correct  locations  are  highlighted. 


The  program  tells  the  user  that  if  the  piece  is  dropped  in  that  location  it  is  an 
illegal  move.  Should  the  user  release  the  mouse  button  at  this  point  the  program 
would  bind  the  location  input  to  an  incorrect  value,  beep  to  present  the  erroi ,  and 
de-activate  the  operation. 

Event:  user  moves  the  mouse  to  adjacent  square. 

lap  it  CC:  plan  to  bind  the  location  input  of  the  move  operation 

to  LOCATION  -VN  DF.R  -MOtSE. 

Output  CC:  the  location  input  of  the  move  operation  is  incorrect. 

Display:  nil.  only  correct  locations  are  highlighted. 
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Event :  user  moves  the  mouse  to  adjacent  square. 

Input  CC:  plan  to  bind  the  location  input  of  the  MOVE  operation 

to  LOC ATION -U NDER-MOU SE. 

Output  CO.  the  location  input  of  the  move  operation  is  correct. 
Display:  highlight  the  location. 


Event:  user  releases  the  mouse  button. 

Input  CC:  bind  the  location  input  of  the  move  operation  to  LOCATION-under-mouse 
execute  the  move  operation. 

Output  CC:  changed  the  piece  input  of  the  MOVE  operation. 

Display:  Erase  the  piece  from  the  old  location  and  display  it  at  the  new  location. 


I 

I 


Should  the  user  move  the  mouse  to  an  adjacent  location  without  releasing  the 
mouse  button,  the  program  would  interpret  the  event  as  a  not-plan  to  bind,  and 
would  remove  the  highlighting  from  the  location. 
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Figure  1:  The  architecture  of  the  chess  program. 

Note  that  even  the  low  level  details  of  the  interface  such  as  displaying  feedback  in 
response  to  individual  mouse  movements  can  be  expressed  in  terms  of  the  communi¬ 
cation  concepts. 

5  Nephew:  A  UIMS  Based  on  Communication  Concepts 

The  communication  concepts  described  above  serve,  not  only  to  explain  the  behavior 
of  graphical  user  interfaces,  but  also  as  the  foundation  of  a  UIMS.  This  section  de¬ 
scribes  Nephew,  a  UIMS  lx  d  on  communication  concepts  [SzekelySS].  Nephew  is 
the  successor  to  the  COUSIN  [HayesSo]  UIMS. 

Figure  1  shows  the  architecture  of  a  typical  Nephew  application,  in  this  case  a 
chess  program  that  behaves  as  described  in  section  ■).  The  program  consists  of  four 
kinds  of  building  blocks,  called  recognizers,  commands,  presenters,  and  t  lie  application 
objects  (not  shown  in  the  figure). 

Recognizers.  A  recognizer  is  parser  for  a  complete  input  gesture  such  as  a  mouse 
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click  or  a  mouse  drag.  A  recognizer  produces  communication  concepts  in  re¬ 
sponse  to  input  events  in  the  gesture,  and  sends  the  concepts  to  a  command  for 
interpretation. 

Commands.  A  command  is  a  communication  concept  interpreter  for  one  specific  op¬ 
eration.  Each  command  receives  communication  concepts  from  recognizers  and 
responds  to  them  producing  output  communication  concepts,  which  it  sends  to 
presenters  to  be  displayed. 

Presenters.  A  presenter  displays  communication  concepts. 


As  shown  in  figure  1,  an  application  implemented  with  Nephew  typically  consists  of 
many  recognizers,  commands  and  presenters.  Quit-command  and  save-command  are 
commands  for  the  quit  and  save  operations  provided  by  the  chess  program.  They  are 
made  accessible  to  the  user  by  displaying  them  with  the  presentation  produced  by 
quit-presenter  and  save-presenter  presenters.  The  quit-recognizer  and  save-recognizer  are 
defined  to  activate  their  commands  when  they  detect  a  click  over  their  presentation. 
For  instance,  if  the  user  clicks  the  mouse  over  the  quit-presenter,  then  quit-recognizer 
sends  an  activate  message  to  quit-command.  Since  the  quit  operations  takes  no  pa¬ 
rameters,  quit-command  responds  to  activate  by  sending  itself  an  execute  message, 
thus  initiating  the  execution  of  the  quit  operation.  The  move-recognizers  and  the 
move-command  implement  the  behavior  discussed  in  section  4.  Each  piece  presenter 
has  a  move-recognizer  to  handle  drag  gestures  that  start  at  the  piece.  While  dragging 
a  piece,  move-recognizer  sends  to  move-command  the  sequence  of  communication  con¬ 
cepts  listed  in  section  4.  The  output  communication  concepts  are  displayed  by  each 
piece-presenter  and  by  board-presenter. 

Nephew  is  implemented  in  Lisp  on  a  Symbolics  Lisp  Machine  using  the  Flavors 
object-oriented  programming  package.  Nephew  provides  predefined  building  blocks 
of  each  kind,  implemented  as  classes,  and  the  job  of  the  interface  implementor  is  to 
assemble1  these  building  blocks  to  construct  a  program.  The  following  subsections 
discuss  the  implementation  in  more  detail. 
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Recognizers 

A  recognizer  is  a  parser  for  a  complete  input  gesture  such  as  a  mouse  click  or  a  mouse 
drag.  Nephew  implements  recognizers  as  classes,  one  class  for  each  kind  of  recognizer. 

Recognizer-Basic  implements  the  behavior  common  to  all  recognizers.  This  includes 
methods  to  turn  recognizers  on  and  off,  methods  to  connect  recognizers  to 
presenters,  and  methods  get  the  input  events  from  a  global  event  queue.  The 
response  to  the  input  events  is  implemented  by  the  subclasses  listed  below. 

Click- Recognizer  implements  the  click  gesture.  The  user  interface  implementor  must 
supply  the  communication  concepts  that  the  recognizer  transmits  when  the  click 
starts  and  terminates. 

Drag-Recognizer  implements  the  drag  gesture.  The  default  Drag- Recognizer  is  pro¬ 
grammed  to  drag  an  outline  of  its  presenter  while  the  gesture  is  in  progress. 
The  implementor  must  provide  the  communication  concepts  that  the  recog¬ 
nizer  transmits  when  the  drag  gesture  begins,  each  time  the  mouse  moves,  and 
when  the  gesture  terminates. 

Dispatch- Recognizer  implements  the  standard  dispatch  behavior  to  distribute  events 
to  other  recognizers.  Instances  of  Dispatch-Recognizer  are  attached  to  presenters 
with  sub-presenters  to  dispatch  events  to  the  presenters  attached  to  the  sub- 
presenters. 

Window-Move-Mixin  is  a  subclass  of  Drag-Recognizer  that  allows  windows  to  be  moved 
by  dragging  them  by  their  title  bar. 

For  example,  in  the  chess  program  implementation  shown  before  in  figure  1,  the  move- 
recognizers  are  instances  of  Drag-Recognizer,  and  quit- recognizer  and  save-recognizer  are 
instances  of  Click- Recognizer.  Instances  of  Dispatch- Recognizer,  not  shown  in  the  fig¬ 
ure,  are  attached  to  board-presenter  and  chess-presenter  to  distribute  the  input  events 
to  the  other  recognizers. 

Commands 

A  command  is  a  communication  concept  interpreter  for  one  specific  operation.  Nephew 
implements  commands  as  classes,  and  implements  input  communication  concepts  as 
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methods  of  command  classes.  The  following  are  the  methods  that  implement  the 
communication  concepts: 

execute:  executes  the  procedure  that  implements  the  operation, 
preview:  invokes  the  procedure  that  implements  the  preview, 
activate:  makes  the  command  active, 
deactivate:  makes  the  command  inactive, 
cancel:  cancels  the  execution  of  an  operation. 

Newly  defined  command  classes  inherit  default  implementations  for  all  of  these  meth¬ 
ods.  Implementors  can  override  some  of  these  methods  to  implement  different  inter¬ 
face  styles. 

In  addition,  a  command  must  provide  the  following  methods  for  each  inpul  of 
an  operation.  Nephew  constructs  default  implementations  for  these  methods  from 
the  operation  declarations,  but  the  user  interface  implementor  can  override  them  to 
provide  non-standard  behavior: 

input  a  command  must  provide  a  method  called  input  corresponding  to  each  opera¬ 
tion  input  called  input.  These  methods  access  the  value  of  inputs  or  their  plan 
component.  For  instance,  the  move-command  used  in  the  chess  program  pro¬ 
vides  methods  called  piece  and  location,  corresponding  to  the  piece  and  location 
parameters  of  the  move  operation. 

set- input:  likewise,  a  command  must  provide  a  method  called  set- input  corresponding 
to  each  operation  input  called  input.  These  methods  set  the  value  of  inputs  or 
their  plan  component.  For  instance,  move-command  provides  methods  set-piece 
and  set-location  to  set  the  command’s  piece  and  location  inputs. 

test- input:  likewise,  a  command  must  provide  a  method  called  test  -input  to  test 
whether  a  value  is  a  legal  input. 

alternatives  -input:  generates  the  set  of  valid  inputs  for  input. 

For  instance,  the  move-command  used  in  the  chess  program  overrides  the  test-piece  and 
test-location  methods  with  predicates  that  define  the  rules  of  chess.  Move-command 
also  overrides  the  alternatives-piece  method  to  return  the  set  of  pieces  that  can  be 


moved  at  any  given  time,  and  the  alternatives-location  method  to  return  the  locations 
to  which  the  selected  piece  can  he  moved.  The  alternatives  methods  are  not  used 
in  the  chess  interface  described  ahove,  but  they  could  be  used  in  an  interface  that 
highlights  the  pieces  that  can  be  moved,  and  the  locations  where  a  selected  piece  can 
be  moved.  No  changes  to  the  chess  application  would  be  needed  to  add  this  feature 
to  the  user  interface. 

Nephew  also  provides  a  library  of  command  classes  that  implement  common  in¬ 
terface  styles: 

Command-Basic  provides  the  behavior  common  to  all  commands  bv  defining  default 
implementations  for  the  methods  listed  above. 

Input-Prompt-Mixin  provides  facilities  to  associate  a  set  of  recognizers  with  each  com¬ 
mand  input,  and  to  switch  on  the  appropriate  recognizers  when  the  application 
program  requests  input.  Switching  the  recognizers  on  will  allow  them  to  receive 
input  events  and  thus  decode  communication  concepts  that  set  the  inputs. 

Hour-Glass-Mixin  automatically  runs  the  hour-glass-recognizer  while  an  operation  is 
executing.  This  recognizer  displays  an  hour-glass  cursor  and  intercepts  all  input 
events. 

Confirmation-Mixin  overrides  the  execute  method.  Before  invoking  the  application 
operation  it  switches  on  a  recognizer  to  prompt  the  user  for  a  confirmation  to 
execute  the  operation. 

For  example,  in  the  chess  program  the  quit-command  is  an  instance  of  a  command  class 
that  includes  Confirmation-Mixin  so  that  the  user  is  asked  to  confirm  before  exiting 
the  chess  program.  The  save-Command  uses  the  Input-Prompt-Mixin  to  prompt  for  the 
file  to  save  the  state  of  the  game. 

Presenters 

A  presenter  displays  communication  concepts.  Nephew  implements  presenters  as 
classes,  one  class  ror  each  particular  way  of  displaying  the  communication  concepts 
for  each  kind  of  object.  Nephew  provides  presenter  classes  to  display  a  variety  of 
generic  application  classes  such  as  lists,  structures  and  array,  and  also  to  display 
commands  and  even  presenters  themselves. 
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In  Nephew  complex  presentations  are  constructed  by  connecting  several  presenter 
instances  to  form  a  tree.  Presenters  with  children  are  called  parent  presenters,  and 
the  children  are  called  sub-presenters.  For  instance,  in  the  chess  example  the  board 
is  a  presenter,  and  the  pieces  are  sub-presenters  of  the  board  presenter. 

The  following  are  the  presenter  classes  in  Nephew’s  library: 

Presenter-Basic  provides  the  basic  facilities  to  link  presenters  to  the  objects  they 
present,  and  all  the  hooks  into  the  graphics  package.  Presenter-Basic  displays 
its  object  as  a  string  identifying  the  type  of  object  ( e.g .  a  chess-piece).  This  is 
useful  in  the  initial  stages  of  the  implementation  of  a  user  interface. 

Borders-Mixin  provides  the  facilities  to  define  the  borders,  background  and  foreground 
of  presenters. 

Window-Mixin  provides  the  facilities  to  attach  a  presenter  to  a  window. 

Structured-Presenter  provides  the  facilities  for  a  presenter  to  have  sub-presenters. 

Record- Presenter,  List-Presenter  are  sub-classes  of  Structured-Presenter  that  can  present 
records  and  lists.  They  provide  the  methods  that  know  how  to  construct  and 
update  sub-presenters  for  their  respective  data  structures. 

Homogeneous- Presenter  is  a  sub-class  of  Structured-Presenter  for  structured  objects 
whose  components  are  all  of  the  same  type.  The  sub-classes  of  Homogeneous- 
Presenter  can  use  a  single  sub-presenter  to  display  all  the  components  of  an  ho¬ 
mogeneous  structured  object,  thus  saving  a  large  amount  of  storage.  Homogeneous- 
Array-Presenter  and  Homogeneous-List-Presenter  are  sub-classes  of  Homogeneous- 
Presenter  specialized  to  arrays  and  lists. 

Rectangular-Alignment-Mixin  provides  definitions  to  align  sub-presenters  in  columns 
or  rows. 

String,  Icon,  Bitmap  and  Color-Presenter  provide  commonly  used  presentations. 

The  chess  program  uses  many  of  these  classes.  For  example,  the  piece- presenters 
are  instances  of  Icon-Presenter,  board-presenter  is  an  instance  of  Homogeneous-Array- 
Presenter.  and  it  uses  the  Borders-Mixin  to  define  the  borders  of  the  board.  The  menu 
containing  the  save  and  quit  commands  is  presented  using  a  List- Presenter  to  present 
the  list  of  save-command  and  quit-command,  and  uses  the  Rectangular-Alignment-Mixin 
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to  align  the  presentations  in  a  column,  left  aligned,  and  with  some  space  between  the 
items. 

Nephew  presenters,  commands  and  recognizers  are  similar  to  Smalltalk’s  MYC 
views  and  controllers  [Goldberg83].  The  main  difference  is  that  the  role  of  the  con¬ 
trollers  is  played  by  commands  and  recognizers  in  Nephew.  By  separating  gesture 
handling  (recognizers)  from  dialogue  control  (commands),  Nephew  simplifies  the  de¬ 
sign  of  the  controllers. 

Presenters,  controllers  and  recognizers  are  also  similar  to  MacApp’s  [Schmucker86] 
views  and  commands.  The  role  of  commands  in  both  systems  very  similar,  serving 
to  collect  the  inputs  for  the  program  operations.  Nephew  takes  the  idea  one  step 
further  by  using  commands  as  an  object  representation  of  the  program  operations, 
and  allowing  a  command  to  be  used  anywhere  where  an  object  can  be  used.  For 
example,  the  commands  can  be  displayed  with  a  presenter. 

6  Classification  and  Separation  In  Intelligent  Interfaces 

This  section  first  gives  a  definition  of  intelligent  interface  and  then  shows  how  commu¬ 
nication  concepts  and  Nephew  could  be  used  to  construct  interfaces  that  are  intelligent 
according  to  the  definition  given  below. 

Intelligent  Interfaces 

When  a  program  communicates  with  the  user  it  has  to  make  certain  decisions,  called 
communication  decisions,  about  the  concepts  it  communicates.  The  program  must 
decide  what  information  to  communicate  to  the  user,  when  to  communicate  it,  and 
how  to  encode  it.  Also,  the  program  must  decode  the  input  from  the  user,  and  then 
interpret  the  decoded  concept. 

A  user  interface  can  be  called  intelligent  in  the  measure  to  which  communication 
decisions  are  conditioned  on  an  analysis  of  the  information  listed  below  [RisslandS2j: 

Program  model:  a  model  of  the  capabilities  of  the  program,  such  as  the  objects  it 
supports,  and  the  operations  it  provides.  Also  included  here  is  a  description 
of  the  program’s  user  interface  design  choices  ( e.g.  whether  the  program  uses  a 
menu  to  display  the  operations). 
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User  model:  a  model  of  the  characteristics  of  the  user  of  the  program,  such  as  his  or 
her  expertise,  preferences  and  past  history  of  interaction  with  the  program. 

Task  model:  a  model  of  the  tasks  the  user  wants  to  accomplish  when  using  the  pro¬ 
gram. 

Workstation  model:  a  model  of  the  characteristics  of  the  workstation  such  as  its  op¬ 
erating  system,  speed,  input  and  output  devices. 

Knowledge  about  interface  design:  knowledge  about  graphic  design,  wording  of  error 
messages,  interaction  styles,  etc. 

For  instance,  an  interface  that  decides  whether  to  use  a  menu  or  a  command  line 
interface  based  on  an  analysis  of  the  user  model  ( e.g .  to  find  out  how  familiar  a  user 
is  with  a  program)  is  more  intelligent  than  one  that  makes  the  decision  irrespective 
of  the  information  contained  in  the  user  model. 

There  are  two  major  ways  in  which  these  models  can  be  used.  The  difference 
comes  from  whether  the  knowledge  is  used  only  at  the  time  the  program  is  designed, 
or  whether  the  program  itself  can  reason  with  this  knowledge  at  run  time. 

When  the  knowledge  is  used  only  at  design  time,  it  is  used  to  make  user  interface 
design  decisions,  which  are  then  hardwired  into  the  specification  of  the  program. 
Software  tools  that  aid  in  incorporating  intelligence  into  a  user  interface  at  design 
time  can  be  called  designer's  assistants. 

When  the  knowledge  is  used  at  run  time  the  program  can  make  user  interface 
design  decisions  tailored  to  the  specific  user  and  specific  interaction  problems  that 
occur  while  the  user  is  interacting  with  a  program.  These  kind  of  interfaces  can  be 
called  adaptive. 

Figure  2  shows  how  interfaces  are  constructed  using  Nephew.  The  interface  de¬ 
signer,  represented  by  the  box  labeled  reasoning  component  chooses  presenters,  com¬ 
mands  and  recognizers  from  the  building  block  library,  and  glues  them  together  using 

Nephew. 

In  Nephew’s  current  implementation,  the  reasoning  component  is  a  human  acting 
at  design  time  rather  than  at  run  time.  A  designer  selects  bui'ding  blocks  from  the 
Nephew  library,  tailors  them  to  a  specific  application,  and  defines  the  connections 
between  them.  Given  the  architecture  of  Nephew,  part  of  the  reasoning  component 
could  be  replaced  by  a  designer's  assistant  and  an  adaptive  interface.  The  designer  s 
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Figure  2:  Constructing  interfaces  with  Nephew. 

assistant  would  help  the  human  designer  choose  and  tailor  the  building  blocks.  The 
adaptive  interface  would  be  part  of  the  Nephew  run  time  environment,  and  it  could 
tailor  and  replace  building  blocks  after  reasoning  about  the  state  of  the  dialogue  with 
the  user.  The  following  subsections  discuss  how  communication  concepts  and  the 
architecture  of  Nephew  facilitates  the  automation  of  the  reasoning  component. 

Revising  Communication  Decisions 

Intelligent  interfaces  operate  by  generating  or  revising  communication  decisions.  A 
designer’s  assistant  generates  communication  decisions  at  design  time,  by  suggesting 
different  input  techniques  and  different  ways  to  display  information.  The  human  de¬ 
signer  selects  from  the  possibilities  offered  by  the  assistant,  and  then  iterates  revising 
the  decisions  before  settling  on  a  design.  The  usefulness  of  such  an  automated  assis¬ 
tant  is  hampered  if  the  different  designs  cannot  be  quickly  implemented  and  tested. 
Separation  allows  replacing  the  modules  that  implement  the  different  designs  with¬ 
out  reprogramming,  so  separation  facilitates  testing  different  designs.  An  adaptive 
interface  places  even  more  stringent  requirements  on  separation  because  the  design 
decisions  are  changed  at  run  time,  and  hence  must  be  made  without  reprogramming. 

For  instance,  consider  the  following  alternative  interface  designs  for  an  operation 
with  a  single  parameter. 

•  The  parameter  gets  the  value  of  the  currently  selected  object,  and  the  operation 
is  chosen  from  a  menu.  The  Macintosh  interface  has  man}'  examples  of  this 
design  ( e.g .  cut  and  paste). 
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•  The  operation  is  presented  as  an  icon  next  or  close  to  the  presentation  of  the 
object  that  it  acts  upon.  Clicking  on  the  operation  icon  invokes  the  operation. 
For  instance,  in  the  Macintosh  interface  the  “close-window”  operation  is  pre¬ 
sented  as  an  icon,  called  the  close  box ,  in  the  top  left  corner  of  the  window  to 
which  it  applies.  Each  w'indow  has  its  own  close  box. 

•  The  operation  is  presented  as  an  icon  and  the  parameter  is  specified  by  dragging 
a  presentation  of  the  object  into  it.  For  example,  the  Macintosh  Finder  presents 
the  “delete-file”  operation  as  a  trash-can  icon.  Files  are  deleted  by  dragging 
their  presentation  to  the  trash  icon. 

In  Nephew  <di  these  interfaces  are  defined  in  terms  of  the  set  of  communication  con¬ 
cepts  discussed  in  section  3.  So,  the  modules  that  implement  the  different  interface 
designs  can  be  plugged  into  the  functionality  portion  of  the  application  without  repro¬ 
gramming.  The  revised  communication  decisions  suggested  by  a  designer's  assistant 
could  be  implemented  automatically  and  tested  immediately  by  the  human  designer. 

Reasoning  About  Building  Blocks 

A  problem  with  many  tool-kits  is  that  it  is  hard  to  find  the  appropriate  building  block 
for  a  given  situation.  Classification  can  be  used  to  construct  a  designer’s  assistant 
to  help  user  interface  designers  to  find  the  right  building  block  and  to  tailor  it  for  a 
given  situation. 

Communication  concepts  can  be  used  to  describe  building  blocks,  providing  a 
powerful  language  for  designer's  assistants  to  index  into  a  database  of  building  blocks. 
A  query  by  example  browsing  too!  such  as  Rabbit  [TouS2]  or  Backbord  [YenSS], 
can  then  be  used  to  find  candidate  building  blocks  given  the  specification  of  a  few 
attributes  of  the  building  block. 

For  instance,  suppose  the  drag  recognizer  building  block  was  described  as  follows: 

The  dragger  can  be  used  to  decode  the  activate  communication  concept. 

It  is  appropriate  for  operations  that  are  presented  as  an  icon. 

It  is  appropriate  when  the  potential  parameter  objects  are  presented  as 
icons. 

It  can  be  used  with  operations  that  require  at  least  one  input. 
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Suppose  the  designer’s  problem  is  to  design  the  input  side  of  the  interface  for  an 
operation  with  a  single  parameter.  The  designer  first  specifies  a  query  for  a  recognizer 
to  activate  an  operation  with  a  single  parameter.  Given  that  many  recognizers  can 
activate  operations,  the  browsing  tool  returns  multiple  choices  ( t.g .  the  designs  for 
single  parameter  operations  discussed  above).  The  designer  can  then  refine  the  query, 
say,  by  specifying  that  both  the  operation  and  the  parameters  will  be  displayed  as 
icons,  and  narrow  down  the  set  of  building  blocks  to  find  the  above  dragger. 

Describing  the  building  blocks  in  terms  of  the  communication  concepts  also  fa¬ 
cilitates  constructing  consistent  interfaces.  For  instance,  if  the  designer  specifies  a 
command  icon  that  presents  the  “command  active”  communication  concept  using 
reverse  video,  then  the  tool  can  easily  detect  an  inconsistency  if  the  designer  specifies 
another  command  icon  that  also  uses  reverse  video  to  present  the  “correct  input” 
communication  concept.  The  designer’s  assistant  can  point  out  the  inconsistency  and 
suggest  ways  to  correct  it. 

7  Final  Remarks 

The  language  of  communication  concepts  described  in  this  paper  has  two  salient 
features: 

•  The  communication  concepts  specify,  in  abstract  terms,  what  information  a 
program  communicates  with  a  user,  without  specifying  how  that  information  is 
communicated. 

•  The  communication  concepts  support  the  transmission  of  all  the  information 
needed  to  implement  a  wide  variety  of  graphical  interfaces. 

The  main  consequence  of  these  two  features  is  that  communication  concepts  can  be 
used  to  define  an  application/user-interface  interface,  that  is,  the  interface  between 
the  application  and  user  interface  modules  of  a  program.  The  language  specifies,  not 
only  what  information  must  be  supplied  by  the  application  module  in  order  to  support 
the  user  interface,  but  also  how  this  information  must  be  encoded. 

This  language  is  good  from  the  modularity  and  code  reusability  point  of  views. 
The  user  interface  can  be  changed  without  affecting  the  application  portion  of  the 
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program,  and  the  interface  building  blocks  can  be  reused  because  they  are  plug- 
compatible  with  the  application  portion  of  the  program.  Such  a  clean  architecture 
is  important  for  the  construction  of  intelligent  interfaces  because  it  facilitates  revis¬ 
ing  user  interface  design  decisions.  Also,  communication  concepts  are  concepts  that 
intelligent  interfaces  can  reason  about  when  making  design  decisions. 
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